Welcome to the real world Beautifulmind! Your present large cong will hardly miss you if don't go.
I remember the joy of not attending meetings, doing pleasurable things other than listening to the cult drones droning on.
I agree with others that it's great you have your spouse with you, my fade cost me my wife and two of my three kids, damn it! But that was some time ago. There is plenty of time to meet new friends who don't have the cultic baggage. Best wishes to both of you and keep us posted.